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ABSTRACT 

An evaluation of 11 university grants funded by the 
Louisiana Department of Education's Office of Special Education 
Services (OSES) during the 1989-90 academic year is presented. The 
methodology used was based on an evaluation format adopted by the 
Louisiana Board of Elementary and Secondary Education for evaluating 
Louisiana Education Quality Support Fund projects. The evaluation 
process included the following step - ?: (1) analysis of grant 
applications; (2) development of interview protocols; (3) site 
visits; (4) completion of Project Objective Management Forms (POMFs); 
(5) collection and analysis of Mid-Year Reports (MYRs) from project 
directors; (6) completion of Project Summary Evaluation Forms based 
on information from the POMFs and MYRs; and (7) production of 
narrative summaries for all projects. This paper includes summaries 
of the formative and summative evaluations as well as a critique of 
the overall OSES process for monitoring and evaluating these 
projects. Results from the summative evaluation were generally 
positive, although there was considerable variance across programs 
and categories. Six areas of weakness were uncovered through the 
formative evaluation, and recommendations were provided in these 
areas. (Author/TJH) 
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Abstract 

This paper describes an evaluation of university grants funded 
by the Louisiana Department of Education Office of Special 
Education Services (OSES) during the 1989-90 academic year. The 
paper focuses on the methodology employed, the findings, and 
recommendations for improving the projects. 

The methodology was based on an evaluation format adopted by 
the Louisiana Board of Elementary and Secondary Education for 
evaluating Louisiana Education Quality Support Fund (LEQSF) 
projects. That evaluation process included the following steps: 

1) analysis of grant applications; 

2) development of interview protocols; 

3) site visits; 

4) completion of Project Objective Management Forms (FOMFs) ; 

5) collection and analysis of Mid-Year Reports (MYRs) from 
project directors; 

6) completion of Project Summary Evaluation Forms (PSEFs) , 
based on information from the POMFs and MYRs; and 

7) production of narrative summaries for all projects. 
This paper includes summaries of the formative and summative 

evaluations, plus a critique of the overall OSES process for 
monitoring and evaluating these projects. Results from the 
summative evaluation were generally positive, although there was 
considerable variance across programs and categories. Six areas of 
weakness were uncovered through the formative evaluation and 
recommendations were presented in these areas. 
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Statewide Evaluation of Eleven Special Education 

University Grants 

Ahis paper describes the evaluation of 11 university grants 
that were funded for a total of $460, 000 by the Louisiana 
Department of Education Office of Special Education Services (OSES) 
during the 1989-90 academic year. 

From the OSES perspective, the evaluation objectives were both 
formative and summative in nature in that they were intended to: 

1) determine the success of each project in meeting its 
goals, 

2) discover and describe unusually effective programs, and 

3) uncover problems encountered by the universities in the 
conduct of their programs and recommend strategies for 
overcoming them. 

The evaluators also saw the study as an opportunity to pilot 
an evaluation system that was adopted in 1989 by the Louisiana 
Beard of Elementary and Secondary Education for statewide use, 
including the evaluation of more than $24 million in Louisiana 
Education Quality Support Fund* (LEQSF) projects. 

The OSES evaluation, which was conducted by staff at Louisiana 
State University in the summer of 1990, drew upon: 

1) a rationale for linking goals to activities to measurable 
outcomes, such as that described by Rutman (1977), and 



2) the evaluation standards developed by the Joint Committee 
on Standards for Education Evaluation (1981). 

Also of concern to the evaluators was the utilization of the 
evaluation results inasmuch as under-utilization of such results 
(Shapiro, 1986) is a chronic problem, especially with state 
agencies (Peck & Triplett, 1983) . 

Methodology 

As mentioned previously, the overall methodology employed for 
evaluating the OSES College/University Grants was patterned after 
the system utilized by BESE for the evaluation of LEQSF projects. 
That overall process involved a number of components: 

1) a review of each project's grant application, 

2) a site visit and interview, 

3) the joint completion of the Project Objective Management 
Form (POMF) by the evaluator(s) and project director 
dur-.ng the site visit, 

4) completion of the Mid-Year Report (MYR) by the project 
director, 

5) completion of the Project Summary Evaluation Form (PSEF) 
by the e valuator (s ) , and 

6) generation of a narrative description of the program by 
the evaluator (s) . 

The narrative was not part of the evaluation procedure adopted 
by BESE but was included in tne evaluation of the OSES 
College/University Projects (K.T. Associates, 1990) to provide a 

4 
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capsule description of each project. A visual overview of the 
process is found in Table 1. 

Grant Proposal 

Upon receipt of the OSES contract in May 1990, the evaluators 
carefully reviewed the most recent grant proposals for the 
projects, nine of which were in their second year of funding. The 
remaining two programs were in their third year of funding. 

In order to generate questions for the interviews and to 
structure information that would becon.e part of the POMF, the 
evaluators listed the objectives identified in the grant proposals, 
the activities that the project managers intended to conduct by way 
of accomplishing their objectives, and the evaluation mechanisms 
they would use to gauge their program's progress. 

It may be parenthetically noted that, in many cases, the 
objectives were not linked to the activities, which in turn were 
not linked to th 5 evaluation measures. This lack of continuity 
caused the evaluators some difficulty in establishing interview 
protocols and in completing the POMFs. 

Qn-Site Interviews 

The grant applications were reviewed prior to the on-site 
interviews, which were conducted during May and June. Each site 
was visited, with multiple-person days spent at each location. At 
each site, the evaluators determined through interviews and 
observations whether modifications had been made to the project 



objectives, activities, and evaluations. Tne evaluators' written 
comments and interview tapes were then used to complete the POMFs. 

Project Objective Management Form (POMP 

After all the interviews had been completed, the evaluators 
filled out the POMFs, recording the objectives, activities, and 
evaluations described in the grant application proposals and noting 
where modifications had been made. Progress was then noted on each 
original objective, activity, and evaluation as well as on those 
that were modified, in some cases (e.g., the Nortneast Louisiana 
University project), no modifications had been made; on others 
(e.g., the Grambling State University project), numerous 
modifications were listed. 

It became obvious during the course of the interviews that the 
project directors' expertise in completing the grant applications 
varied considerably from one project to the next. In many cases, 
the project director and assistant director were quite adept a 
securing grants, but were unfamiliar with the rationale for linking 
objectives, activities, and the evaluation cf outcomes. OSES 
should be more explicit in stressing this linking rationale in 
future grant application processes. 

Mid-Year Reports (MYRsJ 

The Mid-Year Reports were to have been completed before the 
interviewers went into the field. "his happened in most cases, 
although seme pro:ect directors did net complete their MYRs until 
after the site visits. 
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The overall evaluation of the 1990-91 OSES College/University 
Grants was complicated by the fact that several of the projects 
were not completed until the end of June, July, or even August 
1990. Given the July 1990 deadline for completing the overall 
evaluation, some projects were evaluated before their programs were 
completed. 

The project directors were generally sympathetic to the 
evaluators' deadline difficulties and in some cases, simply 
submitted their Mid-Year Reports the first week of July, regardless 
of the status of their projects. Those that did so were sometimes 
forced to turn in partial MYRs, noting that they could not 
1 ligently respond to some of the questions so early in their 
programs . 

Once the POIFs were complete, the- evaluators drafted project 
narratives, drc ring on information from ^he POMFs, the MYRs, the 
original grant proposals, and the site visits. Field notes {which 
were transcribed and edited) and audio tapes made by the evaluators 
offered valuable sources of qualitative data. Though the 
narratives were begun at the end of June 1990, most of the writing 
occurred in July. 

Project Summary Evaluation Forms (PSEFs) 

The evaluators also began work on the Project Summary 
Evaluation Forms (PSZFs) in June, drawing cn information from the 
POMFs, MYRs , and the narratives. Though the FOMF form C-f fared 
little from the version adopted by HESF, the PSEF form war modified 
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considerably to reflect the special characteristics of 
college/university grants as opposed to the elementary/secondary 
grants for which it was originally designed. The original PSEF was 
13 pages long and consisted of eight sections; the version used in 
the OSES evaluation was five pages long, divided into six sections. 
Information included in the open-ended sections of the elementary- 
secondary PSEF was incorporated into the OSES narratives. 

The PSEF was primarily a summative evaluation. Little or no 
formative evaluation was possible given the time constraints within 
which the evaluation had to be conducted and completed. 

Results 

Summative Evaluation 

The overall results from the "Third Party Evaluation of the 
Office of Special Education Services' College and University 
Grants, 1989-90" (hereinafter referred to as the College/University 
Evaluation or CUE) were very positive. As indicated in Table 2, 
the overall mean score or. the PSEF was 26.3 cut c: a possible total 
score of 30. 

There was, of course, variance across colleges/universities on 
their PSEF total and category scores. The two highest total scores 
were achieved by the Northeast Louisiana University and McNeese 
State University projects, both of which received perfect 30 
scores. it should be .noted that these were the enly two projects 
that were in their third ytar of funding; the ether nine projects 
were in their second year. In conversations with the project 
directors at Northeast and McNeese, both indicated that having had 



two prior years of experience had greatly enhanced the efficacy of 
their projects during the third year. 

The college/university scores on the PSEF ranged from 21 (70 
percent of the total possible) to 30. As indicated in Table 2, 
four projects (i.e., Grabbling State University , Louisiana College- 
1/ Louisiana College-II, and Southeastern Louisiana University) 
scored 80 percent or lower on the PSEF. There also was 
considerable variation across PSEF categories. The projects scored 
highest on characteristics of participants (56.7 percent), while 
scoring lowest on adequacy of evaluation design (75.8 percent). 

The variation among scores on the specific PSEF indicators was 
even greater than that in the PSEF categories as indicated in Table 
3. It should be noted that scores from the 31 PSEF indicators were 
used to produce the six PSEF category scores. All PSEF indicators 
must have been rated "four" or "five" on a five-point sc.Ue in 
order for the PSEF category to be rated "excellent" ("three" on a 
three-point scale) . Additionally, if any two or mere PSEF 
indicators were rated "one" or "two" on the five-point scale, the 
category would be rated "unsatisfactory" ("one" on a three-point 
scale). [Copies of the PSEF with categories and indicators will be 
available at the AERA session.] 

As indicated in Table 3, the College/University scores across 
ail PSEF indicators ranged fr~-m 107 (69 percent cf total possible) 
to 14 9 (96.1 percent of total possible) . The moan score across ail 
grants was 131.7 (34.5 percent), with a standard deviation of 13.5. 
Overall scores by indicators within categories ranged f rem highs cf 
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94.2 percent and 90.9 percent for characteristics of participants 
and personnel to lows of 79.2 percent and 7 9.1 percent for adequacy 
of evaluation and attainment of stated purposes/objectives. 

It is noteworthy that the two projects receiving perfect PSEF 
scores utilized two different approaches: the Northeast project 
used the summer institute/workshop approach, while the McNeese 
project used the regular academic course approach. Since four 
other projects used the summer institute/workshop approach and 
another four used the regular academic course approach, these two 
exemplary projects may provide useful models for other projects. 

Formative Evaluation 

Exemplary Programs. Particularly valuable aspects of the 
Northeast Louisiana University Summer Institute were: 

1) the very impressive group of presenters who took part in 
the Institute; 

2) the attractive marketing campaign mounted by the project 
staff; 

3) the vocational education fccus on "ecological congruence" 
between the worker and the workplace, which could be 
transported to other special education units around the 
state; and 

4} the excellent evaluation cf student performance component 
of the Summer Institute. 
The principal investigator at McNeese State University indicated 
that he had integrated seme of Northeast's "ecclcgical congruence" 
literature into his regular academic courses. Such cross- 
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fertilization is an excellent ancillary outcome of the 
College/University grants. 

Particularly valuable aspects of the McNeese program, which as 
previously mentioned took the regular education curriculum 
approach, were; 

1) the alteration of course descriptions/requirements in the 
university catalog, based on innovations resulting from 
the project itself; 

2) the project's emphasis on increasing the vocational- 
technical and job analysis skills of teachers of 
mild/moderate secondary education students; and 

3) the strong evaluation of student performance component of 
the grant. 

The procedure used by the principal investigator to 
institutionalize change in curriculum offerings should be followed 
by other grantees using the regular education curriculum approach. 

Other Programs , Other projects showed exceptional promise, 
including the Louisiana State University-I program and the two 
University cf New Orleans projects. Positive aspects of the LSU-I 
grant included: 

1} the project's emphasis on coordination between the 
university, the state Department cf Education, and local 
education authorities ; 

2) zhB utilization cf teieiearning technologies; and 

3) the development of modules that could be utilized by 
other special educators in the state . 
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The University of New Orleans-II project was particularly 
interesting inasmuch as its major focus was the creation of model 
sites that maximize the degree to which students with severe 
handicaps can be integrated into the regular curriculum. The UNC-I 
grant, on the other hand, showe«u promise of establishing a 
statewide program to train teachers to work with visually impaired 
students. This concept of statewide training is articular 
important inasmuch as many special education teachers have 
difficulty locating suitable course offerings in their regions of 
the state. The evaluators saw great potential in integrating the 
UNO- 1 statewide training concept more fruitfully with the LSU-I 
-ning approach. 

The projects did, of course, experience some problems during 
*w..a 1989-90 funding cycle. One would expect seme difficulties in 
the administration of a program that consisted of more t>an 
$460,000 in projects operating at 11 separate sites across n 
entire program year. The difficulties that were noted include: 

1) inadequate linking of goals zo activities to evaluations 
by project directors; 

2) management problems at the project sites cue tc loss of 
key personnel; 

3) generally less than excellent evaluation procedures at 
some project sites; 

•1) inadequate monitoring by the stzite Department of 
Education staff due to manpower shortages in that agency; 

12 
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5) lack of a uniform system for determining the progress 
that program participants are making toward new or 
expanded certification in special education; and 

6) lack of coordination and communication among the 
College/University projects. 

Before presenting recommendations for addressing these 
problems, a note should be made about the CUE. The scope of the 
CUE wjf? limited by virtue of the fact that the evaluators did not 
receive a final funding decision until May 1990 and the site visits 
had to be completed by July 6. The major problem caused by this 
schedule was the fact that preparations for the five summer 
institutes were not complete during the time of the site visits. 
This, of course, impeded the evaluation process. 

Repercussions of the timeframe constraints included the 
following . 

1) No meaningful formative evaluation could occ*,*, .inasmuch 
as the timeframe for the evaluation was so short. 

2) Some project directors were defensive during the site 
visits because they were busy preparing for or just 
starting their summer institutes. 

3) Some project directors were unsure how to complete their 
Mid-Year Reports inasmuch as their major activity - their 
summer institute - was just getting underway. 

The CUE scope was further limited by the fact that the 
evaluation was funded at approximately 1.4 percent of the tctal 
value ox the projects. As a rule of thumb, an evaluation with 
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complete formative and summative evaluation requires funding at 3-5 
percent of the total project expenditure. The evaluation should 
have started in October 1989 rather than May 1990 with a 3-5 
percent budget, which would have allowed sufficient time and 
resources to stage site visits early in the program year. Such 
visits would have greatly enhanced the formative evaluation. 

Conclusions and Recommendations 

The conclusions and recommendations section will center around 
six problem areas identified in the previous section. 

Inadequate l inkages 

As mentioned previously, several project directors 
inadequately linked their project goals, activities, and 
evaluations. To assist them in the future, the evaluators 
recor-. tended that OSES consider holding a training session for all 
project directors in which the linking rationale is explained. 
Short of this, OSES was advised tc send each College/University 
expressing an interest in applying for funds a written explanation 
of the importance of linking objectives to activities that are in 
turn linked to a set of evaluations. 

If all funded projects had such a linkage, the chances of 
their being both successful and evaiuable would be enhanced. A 
linking mechanism helps the grantee to organize his/her project 
activities and evaluations, and it heirs the third par„y evaiuator 
to determine if the project is working (summative evaluation) or, 
if not, what can be done (formative evaluation) . 
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Personnel Turnover 

At least three projects (i.e., the Southeastern Louisiana 
University, Louisiana College-II, and Louisiana State University-I 
programs) lost key personnel or consultants during the year, thus 
hampering their potential success. While the LSU-I project was 
abl* to rectify the problem, the SLU and Louisiana College projects 
suifered timeline and evaluation problems. 

LDE Manpower Shortages 

The problem of project staff turnover was exacerbated by a 
related problem: the inadequate monitoring of projects by OSES 
staff due to manpower shortages within the state Department of 
Education. How could OSES address the associated problems of 
personnel attrition at both the university and state levels? 

One recommendation was to give a Program Manager at the 
Department of Education specific part-time responsibility for 
monitoring the College/University grants throughout the program 
year. (Oversight of the projects was a shared responsibility among 
several individuals, with no single staff person taking ultimate 
responsibility) . Should that OSES staff person change jobs, the 
evaluators recommended that another staff person be immediately 
assigned those monitoring responsibilities. 

If the OSES staff maintained regular contacts with the project 
directors through the program year, they would become immediately 
aware of personnel losses at the local level and could help the 
Cciiege/University involved respond by locating replacement staff 
cr developing other coping strategies. An alternative strategy 
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would be to fund future evaluations at the 3-5 percent level and 
expect the third-party evaluator to monitor the projects throughout 
the year as part of the formative evaluation process. 

Program Evaluation Problems 

With regard to problems with the individual project 
evaluations, some difficulties would be addressed if evaluations 
ware properly linked to objectives and activities. The evaluators 
recommended that the OSES staff also consider providing all grant 
recipients in-service training on evaluation techniques. A one-day 
seminar for all project directors would be both useful and cost- 
effective. The evaluators nonetheless pointed out that OSES might 
have to make attendance mandatory to ensure full participation, 
inasmuch as some College/University personnel might either assume 
they already were knowledgeable abuut evaluations or might not see 
the worth of proper evaluation strategies. 

Certif icatio - .i Tracking 

Another problem uncovered by the CUE was the lack of a uniform 
system for determining how many program participants had either 
gained or expanded their certification as a result of having 
attended the classes, institutes, and workshops sponsored by the 
various projects. Several grantees listed this as a major project 
objective, and OSES requests certification data as part of the MYR. 
Despite this, no project director had an adequate method cf 
tracking students' progress toward certification. Several project: 
directors indicated that the appropriate data collection point was 
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the state Department of Education inasmuch as it is the state 
teacher certification agent. 

Obviously, solving the problem would require a joint effort by 
OSES and the College/University grantees. Each grantee should 
maintain an up-to-date cumulative file of all participants with 
Social Security numbers, current certification status, and type of 
certification sought. This data should be forwarded semiannually 
to staff at OSES; they in turn should work with the department's 
Bureau of Teacher Certification and computer section to set up a 
system whereby participant files are cross-matched with 
certification files. 

A computerized tracking system would answer the important 
question of how many teachers have become certified in special 
education as a result of the College/University projects. By 
comparing this information with data from Teacher Certification, 
pinpointing manpower needs around the state in the various special 
education specializations, OSES staff could help present and future 
grant applicants keep abreast of changing manpower needs. 

Lack of Coordination 

The final problem area has to do with the lack of coordination 
and communication among College/University projects. For instance, 
five summer institutes were held in 1990, but only one (Louisiana 
College-I) was conducted in cooperation with other universities. 
Such joint ventures are of great benefit if, as in the case cf 
Louisiana College-:, they make it possible to expand programming to 
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several sites around the state. Unfortunately, lack of 
communication and coordination among campuses is characteristic of 
Louisiana higher education; in fact, Governor Buddy Roemer often 
refers to the state's "Lebanon of higher education." 

In conversations with OSES, the evaluators suggested that the 
agency strongly encourage inter-institutional cooperation in the 
future. Consortium funding arrangements might be one way to foster 
such cooperation. Another strategy would be to have project 
directors share their "Best Practices." Perhaps the best strategy 
would be for OSES to call a project directors workshop at the end 
of the funding period, during which each director would present a 
final report to their colleagues and to the OSES staff. 

Ancillary Findings 

As mentioned previously, the under-utilization of findings is 
a chronic problem in the evaluation of public programs (Shapiro, 
1986; Peck & Triplett, 1983) and was viewed by the evaluators 
as a probable outcome of the 198 9-90 CUE. 

Considerations of Staffing 

The decision to undertake a third party evaluation of the OSES 
College/University projects was initiated by the then-OSES program 
manager, who convinced her supervisors that such an overall 
evaluation was necessary. When she later accepted a new job, no 
other staff person assumed primary responsibility for the 
College/University grants, prompting the -.valuators' inclusion of 
Recommendation 4 in the 1989-90 CUE report. 

1 Q 
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Under the circumstances, the evaluators became concerned that 
the evaluation findings would be under-utilized due to four 
factors. 

1) Because no program manager had direct responsibility for 
the project, no one was in a clear position to instigate 
action on the evaluation recommendations. 

2) Upper level management at OSES was overworked and might 
not have had time to attend to ;in evaluation follow-up. 

3) The College/University Grants program was entering a new 
funding cycle in which programmatic emphasis was likely 
to change dramatically. An overworked staff might have 
written off the evaluation report rather than 
incorporate/adapt its findings to future programs. 

4) Two of the 11 projects evaluated were in their third year 
of funding and were barred by program guidelines from 
seeking a fourth year of funding. Inasmuch as the 
evaluation took place at the end of the projects' last 
year, the findings would have had no formative value for 
those two projects. What's more, because 
communication/coordination among projects was so poor, it 
also was unlikely that insights from the two third-year 
evaluations would have been shared with the other project 
directors . 

Though the nine projects in their next-to-last year of 
funding could have benefitted from the findings the 
following year, the previously mentioned lack of a "point 
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person" at OSES brought even this potential benefit into 
question . 

Considerations of Timing 

As noted throughout this report, considerations of time and 
timing proved a major impediment both to the conduct of the 
evaluation and to the utilization of its findings. Though the 
evaluation format offered ample opportunity for formative 
evaluation, such analysis was all but impossible within the 
allotted time frame. 

Ideally, the projects should have been evaluated annually, 
beginning with the first year of funding. Even so, noticeable 
improvements in both the conduct of the study and the utilization 
of findings could have been realized, however, had an evaluator 
been hired at the start of the 1939-90 program year rather than at 
its end. 

Had the evaluators reviewed the grant proposals at the start 
of the year, the problem of inadequate goal/activity/evaluation 
linkages could have been addressed early on. Inasmuch as some 
managers found the linking rationale instructive in terms of 
project management, their exposure to such concepts early on might 
have helped them improve project operations. Finally, had the 
formative evaluation commenced in the fall, the evaluators also 
might have been able to alert the OSES staff to the problem of 
staff turnover at various project sites. 

Timing was a problem net only with the star- but the end cf 
the evaluation process: the deadline for submitting the final 
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evaluation report was well in advance of the conclusion of several 
projects. As mentioned previously, the evaluators were required to 
submit final reports at the end of July so that their findings 
could inform the 1990-91 grant application process, which began in 
August . 

Admittedly, the findings should have informed the 1990-91 
review process (assuming that, in the absence of an OSES project 
manager, they were read and digested) . However, the net result was 
that the five summer institutes - all of which were eligible for 
funding in 1990-91 - were still underway at the study's conclusion 
and hence were never evaluated in their entirety . 

It would have been better had OSES required that the 
evaluators submit partial evaluations on the five summer institutes 
in advance of the 1990-91 grant review process, and submit true 
summative evaluations on the completed projects in late August 
Decisions whether to fand the summer institutes in 1990-91 could 
have been made contingent on the rece:'pt of a final favorable 
evaluation and/or the project director's commitment to address 
weaknesses uncovered in the evaluation prior to the start of the 
1990-91 institutes some nine months later. 

At any rate, both the project directors and the OSES staff 
would have profited from having access to summative evaluations of 
the completed programs. With manpower at the Department of 
Education so short, it is highly unlikely that, without a third 
party to compile and digest the summative data into an easily 



digestible format, any OSES staff person would have been in a 
position to do so. 

Considerations of Funding 

As mentioned previously, the evaluation was funded at 1.4 
percent of the total program budget as compared to the 3-5 percent 
funding level typically recommended for full format ive/summative 
evaluations. The intent was not to scrimp; rather the level of 
funding was dictated by the same budgetary constraints that 
compelled the OSES staff to postpone evaluation of any kind until 
some projects were in their third year. 

The net effect, however unintentional, was to severely 
restrict the amount of formative review that could be conducted in 
the way of third-party site observations and interviews or other 
labor-intensive but nonetheless valuable methods of qualitative 
evaluation. it also forced the evaluatcrs to focus closely on 
linkages between goals, activities, and evaluations as the most 
efficient means of quickly familiarizing themselves with highly 
diverse projects and then assessing their equally varied outputs. 

Though dedicated proponents of Scriven's "goal-free 
evaluation" might consider the methodology used tc be too focused, 
the primary evaluation tools - the FCMFs, interview/observation 
protocols, and PSESs - were expressly structured to identify and 
assess latent and emerging objectives and outcomes. As such, the 
evaluators felt that they operated well within the standards 
espoused ty Cronbach (1982) : that they were open tc all ends toward 
which the projects might reascr.abiv aim without clcsir.c cut 
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insights that could be gained from the project staff, and that they 
used their evaluation training to introduce values that otherwise 
would not have been voiced. 

Though additional "goal-free" evaluation could enhance future 
College/University Project evaluations, it is unlikely that the 
OSES budget could support such time-consuming and labor-intensive 
analysis by a third party evaluator. Popham's recommendation (1988) 
that, in the face of budgetary constraint internal "goal-free" 
evaluations be combined with external "goal-based" evaluations, 
seems a possible alternative. 

Though OSES staff could hardly conduct "goal-free" evaluations 
given their familiarity with the projects, they could conceivably 
arrange to trade off evaluation services with one of the many other 
Department of Education units whose programs must also be 
evaluated. For instance, the LEQSF program staff (whose BESE- 
adopted evaluation plan served as the model for the OSES plan) 
could conduct an internal "goal-free" analysis of 
College/University projects in return for the OSES staff's 
evaluation of some of their projects. Again, such a pact would be 
contingent on the ability of the two units ~ both of which are 
chronically under-staffed - to free up the manpower to undertake 
such a venture. 
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